Vehicular control system

ABSTRACT

A vehicular control system includes a central control module vehicle, a plurality of vehicular cameras disposed at a vehicle and viewing exterior of the vehicle, and a plurality of radar sensors disposed at the vehicle and sensing exterior of the vehicle. The plurality of vehicular cameras includes at least (i) a forward-viewing vehicular camera, (ii) a driver side sideward-viewing vehicular camera and (iii) a passenger side sideward-viewing vehicular camera. The central control module receives vehicle data relating to operation of the vehicle. During a driving maneuver of the equipped vehicle, and responsive at least in part to fusion at the central control module of (i) image data captured by at least the forward-viewing vehicular camera and (ii) radar data captured by at least the front radar sensor, a pedestrian present exterior of the equipped vehicle is detected.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a division of U.S. patent application Ser.No. 17/656,069, filed Mar. 23, 2022, now U.S. Pat. No. 11,763,573, whichis a continuation of U.S. patent application Ser. No. 16/949,812, filedNov. 16, 2020, now U.S. Pat. No. 11,288,888, which is a continuation ofU.S. patent application Ser. No. 15/911,417, filed Mar. 5, 2018, nowU.S. Pat. No. 10,839,233, which is a continuation of U.S. patentapplication Ser. No. 14/845,830, filed Sep. 4, 2015, now U.S. Pat. No.9,911,050, which is a continuation of U.S. patent application Ser. No.13/202,005, filed Aug. 17, 2011, now U.S. Pat. No. 9,126,525, which is a371 national phase application of PCT Application No. PCT/US2010/025545,filed Feb. 25, 2010, which claims the benefit of U.S. provisionalapplications, Ser. No. 61/180,257, filed May 21, 2009, Ser. No.61/174,596, filed May 1, 2009, and Ser. No. 61/156,184, filed Feb. 27,2009, which are hereby incorporated herein by reference in theirentireties.

FIELD OF THE INVENTION

The present invention relates to rear vision systems for vehicles and,more particularly, to rear vision systems that provide an alert to thedriver of a vehicle that an object is detected rearward of the vehicleduring a reverse travelling maneuver.

BACKGROUND TO THE INVENTION

It is well known that the act of reversing a vehicle, such as backingout of a garage or driveway or parking space, can be dangerous,particularly if a child or pet wanders behind the vehicle before orduring the reversing process. A variety of backup assist systems areknown to assist the driver in detecting and/or seeing an object in therearward path of the vehicle. For example, rear vision systems are knownthat capture images of a scene rearward of a vehicle (such as during areverse driving maneuver) and display the images of the rearward sceneto the driver of the vehicle to assist the driver during a reversedriving maneuver. Such systems may include a rearward facing camera orimage sensor for capturing images of the rearward scene, and may includeother types of sensors, such as ultrasonic sensors or radar sensors orthe like, which may provide a distance sensing function or proximitysensing function. Examples of such vision and/or imaging systems aredescribed in U.S. Pat. Nos. 5,550,677; 5,877,897; 6,498,620; 5,796,094;6,396,397; 6,806,452; 6,690,268; 7,005,974; 7,123,168; 7,004,606;6,946,978; 7,038,577; 6,353,392; 6,320,176; 6,313,454 and 6,824,281,which are all hereby incorporated herein by reference in theirentireties.

Studies have indicated that drivers, during a reversing act, spend atleast about percent or thereabouts of the reverse driving time lookingover their right shoulder (for left hand drive vehicles, such as used inthe United States), and about 35 percent or thereabouts of the reversedriving time glancing at the rearview mirrors of the vehicle. Thestudies further indicate that, of the 35 percent of time the driver isviewing the rearview mirrors, the driver typically spends about 15percent of that time viewing or glancing at the driver side exteriorrearview mirror of the vehicle, about 15 percent of that time viewing orglancing at the passenger side exterior rearview mirror of the vehicle,and about 5 percent of that time viewing or glancing at the interiorrearview mirror of the vehicle. In spite of the presence of known backupassist systems and rear vision systems, accidents still occur.

SUMMARY OF THE INVENTION

The present invention provides an alert or prompting system for avehicle equipped with a rear vision system comprising a rear mounted andrear viewing imaging sensor and a video display located in an interiorcabin of the vehicle for displaying video images (captured by the rearviewing imaging sensor or camera) to a driver normally operating thevehicle of a scene occurring rearward of the vehicle when the driver isexecuting a backup reverse maneuver. The alert system alerts the driverof a vehicle that an object is potentially rearward of the vehicleduring a backup maneuver. The alert system provides a visual alertdevice preferably at two or more of the rearview mirrors of the vehicleand alerts the driver that an object has potentially been detectedrearward of the vehicle so the driver is alerted or prompted to checkand verify the rearward scene on the video display screen that isdisplaying the images captured by the rear mounted camera or the likewhile backing up the vehicle, and thus the driver is alerted or promptedto verify whether or not the object that has been at least provisionallybeing detected presents a hazard or obstacle. The alert system mayprovide a visual alert device at each or all of the rearview mirrorassemblies of the vehicle, such as at an interior rearview mirrorassembly, a driver side exterior rearview mirror assembly and apassenger side exterior rearview mirror assembly.

During a reversing maneuver (such as when the driver selects a reversegear of the vehicle and before and during reverse travel of the vehicleand while the reverse gear of the vehicle is selected), the driver ofthe reversing vehicle will likely glance at one of the mirror assembliesand thus will likely and readily notice the visible alert when thesystem is detecting an object (such as via machine vision processing ofvideo image data captured by the rear mounted and rear viewing backupcamera itself and/or responsive to non-vision sensors such as a radarsensor or an array of radar sensors and/or such as an ultrasonic sensoror an array of ultrasonic sensors and/or such as an infraredtime-of-flight sensor or array of infrared time-of-flight sensors) andthe alert devices are thus activated, and thus the driver will recognizethat an object has been at least potentially or provisionally detectedrearward of the vehicle and will know to or be prompted to look at thevideo display screen to determine by checking or viewing the displayedvideo images what the object is and where the object is located relativeto the reversing vehicle and whether the object detected presents apotential hazard or obstacle in the contemplated rearward path of travelof the vehicle. Optionally, graphic overlays, such as static or dynamicgraphic overlays (such as graphic overlays of the types described inU.S. Pat. Nos. 5,670,935; 6,222,447 and 6,611,202; and/or PCTApplication No. PCT/US08/76022, filed Sep. 11, 2008 and published Mar.19, 2009 as International Publication No. WO 2009/036176, which arehereby incorporated herein by reference in their entireties) may augmentand aid such determination by the driver.

These and other objects, advantages, purposes and features of thepresent invention will become apparent upon review of the followingspecification in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a plan view of a vehicle incorporating an alert system inaccordance with the present invention;

FIG. 2 is a forward facing view with respect to the direction of travelof the vehicle, showing each of the rearview mirrors of the vehicle ofFIG. 1 and the alert displayed at the rearview mirrors when an object isdetected behind the vehicle during a backup maneuver;

FIG. 3 is a side elevation of a vehicle equipped with an alert system ofthe present invention and a forward facing vision system;

FIG. 4 is a side elevation of a vehicle equipped with an imaging orvision system and display in accordance with the present invention;

FIG. 5A is a view of a displayed image as captured by a forward facingcamera of the vehicle of FIG. 4 ;

FIG. 5B is a view of a displayed image that is derived from image dataof the displayed image of FIG. 5A, as processed to provide sidewardviews at a cross-traffic situation;

FIG. 6A is a plan view of the vehicle showing the area encompassed bydisplayed image of FIG. 5A;

FIG. 6B is a plan view of the vehicle showing the areas encompassed bythe displayed images of FIG. 5B;

FIG. 7 is a plan view of the equipped vehicle showing the areasencompassed by forward, rearward and sideward cameras of the vehicle;

FIG. 8 is a display showing an image captured by a forward or rearwardfacing camera of the equipped vehicle and a “bird view” of the vehicle;

FIG. 9 is a schematic of an imaging or vision or detection system inaccordance with the present invention;

FIG. 10 is a schematic of an RGB-Z sensing system suitable for use withthe vision and/or alert system of the present invention;

FIG. 11 is a schematic of an active safety and sensing system inaccordance with the present invention;

FIG. 12 is a schematic of various sub-systems of the active safety andsensing system of FIG. 11 ;

FIG. 13 is a chart showing relationships between sub-systems of theactive safety and sensing system of FIG. 11 ;

FIG. 14 is a schematic of the interface between the lane departurewarning system and the lane keep assist system of the active safety andsensing system of FIG. 11 ;

FIG. 15 shows a headlamp control feature suitable for use with theactive safety and sensing system of FIG. 11 ;

FIGS. 16, 16A and 16B are a schematic of an image processing systemarchitecture suitable for use with the active safety and sensing systemof FIG. 11 ;

FIG. 17 is a schematic of a system architecture suitable for use withthe active safety and sensing system of the present invention; and

FIG. 18 is a schematic of an image sensor chip suitable for use with theactive safety and sensing system of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to the drawings and the illustrative embodiments depictedtherein, an alert or prompting system 10 for a vehicle 12 includes acontrol or processor 14 for determining if an object may be behind thevehicle and/or in the rearward path of the vehicle, and an alert device16 a, 16 b, 16 c at each of the rearview mirror assemblies 18 a, 18 b,18 c of the vehicle. The vehicle includes a rear vision system thatincludes a display (such as a video display screen for displaying videoimages) within a cabin of the vehicle 12 (such as at the instrumentpanel of the vehicle or such as at an interior rearview mirror assemblyof the vehicle or the like) for viewing by the driver of the vehicle,and a rearward facing camera 22 for capturing images of the scenerearward of the vehicle during a reversing maneuver (such as at orbefore or after commencement of rearward motion of the vehicle, such asresponsive to the driver of the vehicle shifting the gear actuator orselector into a reverse gear position). The video display 20 provides avideo image of a scene rearward of the vehicle, such as captured by therearward facing camera 22. The alert system is operable to actuate atleast one of the alert devices 16 a-c, and optionally and desirably eachof the alert devices 16 a-c, when an object is detected rearward of thevehicle and during a reversing maneuver so that the driver, whenglancing at one of the rearview mirrors 18 a-c, is alerted to thepresence of an object rearward of the vehicle, and can then look at orview or check the display 20 to see what the detected object is andwhere the detected object is relative to the vehicle. The alert system10 thus enhances the driver's awareness of objects rearward of thevehicle to assist the driver during the reversing maneuver, as discussedin detail below.

In the illustrated embodiment, the alert system 10 is operable toactivate the alert devices 16 a-c to generate a visible or visual alertpreferably at all three rearview mirror assemblies (for example, theinterior rearview mirror assembly 18 a, the left or driver side exteriorrearview mirror assembly 18 b, and the right or passenger side exteriorrearview mirror assembly 18 c) or any subset thereof, so that the driverwill likely view or glance at at least one of the visible alerts duringa reversing maneuver (because a driver typically spends about 35 percentof the time during a reversing maneuver looking at or viewing a rearviewmirror assembly). Upon viewing the visible or visual alert at one ormore of the mirror assemblies, the driver is alerted or at leastprompted to the potential of an object rearward of the vehicle and thusknows to check or is prompted to check the video display (such as acenter stack video display at the instrument panel or center console ofthe vehicle or a video mirror display at the interior rearview mirrorassembly of the vehicle) so that the driver can readily see what theobject is and where the object is located with respect to the vehicle bylooking at the video display. Optionally, the alert system may includean alert device at each of the exterior rearview mirror assemblies (andnot at the interior rearview mirror assembly) to provide an alert to thedriver at locations where a driver typically views about 30 percent ofthe time during a backup maneuver.

Although an additional audible alert may be provided when an object isdetected, this is not as desirable as a visible or visual alert sincedrivers are typically less tolerant of audible alerts and more tolerantof visible alerts or displays. Also, such audible alerts may oftenprovide an alert for something that the driver may consider to be afalse alarm. A visual alert is more benign and more tolerated to typicaldrivers than an audible alert and thus the visual prompts of the presentinvention may be displayed at a different threshold of sensitivity. Forexample, typically an audible alert is activated only when the certaintyof an accurate detection is at a higher predetermined threshold level,while a visual alert may be displayed at a lower threshold level ofcertainty. The present invention provides visible alerts and providesthem at locations where a driver is likely to look during a reversingmaneuver (such as before backing up and upon shifting the vehicle gearactuator into a reverse gear position and during the backing up of thevehicle), in order to enhance the likelihood that the driver will bemade aware of a detected object during the reversing maneuver, without apotentially annoying audible alert. Thus, the processor or controllermay set a lower detection confidence threshold at which the visualalerts are activated or illuminated and can optionally and preferablyreinforce, but at a higher detection confidence threshold, with anaudible alert.

The alert devices 16 a-c may comprise any suitable visible alertdevices, such as iconistic displays or the like, and may be disposedbehind the reflective element of the respective mirror assembly so thatthe alert device, when activated, is viewed by the driver of the vehiclethrough the reflective element of the side exterior mirror or interiormirror of the vehicle. Optionally, and desirably, the alert devices maycomprise display-on-demand display devices that are viewable throughpartially reflecting and partially transmissive mirror reflectors oftransflective reflective elements of the mirror assemblies (such astransflective reflective elements of the types described below). Thus,the alert devices are viewable and discernible to the driver of thevehicle when activated, but are substantially hidden or rendered covertor non-discernible (behind the reflective element) when not activated.Such alert devices thus may be provided as large area displays that arereadily visible and viewable when activated, to further enhance thedriver's awareness of the alert device when it is activated.

Optionally, the visual alert devices may be disposed behind thereflective element of the mirror assemblies and viewed through anaperture or window established through the mirror reflector (such as vialaser ablation or etching of the metallic mirror reflector). Optionally,the visual alert devices may be disposed elsewhere and not necessarilybehind the reflective element, such as at a housing or bezel portion ofthe mirror assemblies (such as by utilizing aspects of the indicatorsand displays described in U.S. Pat. No. 7,492,281, which is herebyincorporated herein by reference in its entirety) or elsewhere at or inthe vehicle and not in the mirror assembly, such as at an A-pillar ofthe vehicle and near an associated exterior mirror, such that the visualalert device can readily be seen by the driver of the vehicle when he orshe is glancing at the mirror assembly. Optionally, additional alertdevices may be disposed at other locations on or in the vehicle where adriver may readily view them during the reversing maneuver so as to beprompted to check or look at or view the video display of the rearwardscene behind the vehicle.

Optionally, the alert devices or indicators, such as for the exteriorrearview mirror assemblies, may utilize aspects of blind spot indicatorsor the like, such as indicators or light modules of the types describedin U.S. Pat. Nos. 7,492,281; 6,227,689; 6,582,109; 5,371,659; 5,497,306;5,669,699; 5,823,654; 6,176,602; 6,276,821; 6,198,409; 5,929,786 and5,786,772, and/or U.S. patent application Ser. No. 11/226,628, filedSep. 14, 2005 and published Mar. 23, 2006 as U.S. Publication No. US2006-0061008; Ser. No. 11/520,193, filed Sep. 13, 2006, now U.S. Pat.No. 7,581,859; and/or Ser. No. 11/912,576, filed Oct. 25, 2007, now U.S.Pat. No. 7,626,749, and/or PCT Application No. PCT/US2006/026148, filedJul. 5, 2006, and/or PCT Application No. PCT/US07/82099, filed Oct. 22,2007, and/or PCT Application No. PCT/US2006/018567, filed May 16, 2006and published Nov. 23, 2006 as International Publication No. WO2006/124682, which are hereby incorporated herein by reference in theirentireties.

Optionally, the visible alert devices may flash or change color orotherwise function in a manner that further draws the driver's attentionto the alert so the driver is quickly made aware of the potential hazardrearward of the vehicle so the driver knows to view the video display.Optionally, the visible alert devices may provide different stages ofalerts, and may be adjusted or modulated or altered in response to adegree of danger or proximity to the detected object. For example, thealert devices may initially, upon detection of an object several feetfrom the vehicle, be activated to provide a constant display, and as thevehicle further approaches the detected object, the alert devices mayintermittently displayed or flashed and/or the intensity of the alertdevices may be increased and/or the alert devices may otherwise beadjusted to enhance the viewability of the display.

Optionally, one of the alert devices may be adjusted relative to theother two alert devices to indicate to the driver the general locationof the detected object. For example, if the object is detected towardone side of the vehicle's path, the alert device at the side mirrorassembly at that side of the vehicle may be displayed at a greaterintensity or may be flashed to further indicate to the driver thegeneral location of the detected object. The other two alert deviceswould still be activated so that the driver will have a greater chanceof noticing the visible alert during the reversing maneuver.

Optionally, the image processor may process the image data to detect anobject and/or to classify a detected object (such as to determine if theobject is an object of interest or of a particular type orclassification, such as a person or the like) and/or to determine adistance to the detected object and/or to detect or determine othercharacteristics of the object. Optionally, the system may includenon-vision sensors 24 (FIG. 1 ), such as an ultrasonic sensor/array orradar sensor/array or an infrared object detection sensor/array or thelike, to detect an object and/or to determine a distance to or proximityof a detected object. The alert system may be responsive to either theimage processor or other processor that detects objects either viaprocessing of the captured image data or processing of the outputs ofone or more non-vision sensors at the rear of the vehicle.

Thus, during a reversing operation or maneuver of the vehicle, thedriver may move the gear actuator or shifter from a “Park” position to a“Reverse” position to commence the reversing maneuver. When the gearactuator is moved to the reverse position, the rearward facing cameraand video display may be actuated to display the rearward scene forviewing by the driver of the vehicle. A processor may process thecaptured image data or output of a rearward detecting sensor to detectobjects present in the scene rearward of the vehicle, and the alertsystem may activate the alert devices responsive to an object beingdetected. Because during a typical reversing maneuver, a driver maytypically spend about 35 percent of his or her time viewing one or moreof the rearview mirror assemblies of the vehicle, the alert system ofthe present invention (which may provide a visible alert at each of therearview mirror assemblies) substantially increases the likelihood thatthe driver will be made aware of an object detected rearward of thevehicle during the maneuvering process, and will channel or funnel thedriver's attention to check the video screen to see what the detectedobject is and where it is located relative to the vehicle and thevehicle's rearward path of travel.

Optionally, the output of the rearward facing sensor or system may beprocessed by an image processor, such as, for example, an EYEQ™ imageprocessing chip available from Mobileye Vision Technologies Ltd. ofJerusalem, Israel. Such image processors include object detectionsoftware (such as the types described in U.S. Pat. No. 7,038,577 and/orSer. No. 11/315,675, filed Dec. 22, 2005, now U.S. Pat. No. 7,720,580,which are hereby incorporated herein by reference in their entireties),and analyze image data to detect objects. The image processor or controlmay determine if a potentially hazardous condition (such as an object orvehicle or person or child or the like) may exist in the rearwardvehicle path and the alert system may, responsive to the processor,generate an alert signal (such as by actuation of the visual indicatorsor alert devices 16 a-c and/or an audible indicator or by anenhancement/overlay on a video display screen that is showing a videoimage to the driver of what the night vision sensor/camera is seeing) toprompt/alert the driver of a potential hazard as needed or appropriate.The alert devices thus may provide an episodal alert so that thedriver's attention is channeled or funneled toward the video displaywhen there is a potential hazard detected.

Optionally, the imaging device and/or control circuitry or processor maybe part of or share components or circuitry with other image or imagingor vision systems of the vehicle, such as headlamp control systemsand/or rain sensing systems and/or cabin monitoring systems and/or thelike. For example, the vehicle equipped with the rearvision system andalert system discussed above may also include a forward vision-basedsystem 26 having a forward facing camera (such as at the interiorrearview mirror assembly and/or an accessory module or windshieldelectronics module of the vehicle) and an image processor, such as foruse as a lane departure warning system, a headlamp control system, arain sensor system, an adaptive cruise control system, and/or the like.Because forward and rearward travel of the vehicle are mutuallyexclusive events and are determined by the movement or shifting of thegear actuator or shifter to the selected forward or reverse gear, theimage processor circuitry may be shared by the forward vision system andthe rearward vision system. The processor function thus may switch fromprocessing image data captured by the forward facing camera (when thevehicle is in a forward gear) to processing image data captured by therearward facing camera (when the vehicle is in a reverse gear). Thus,the processing circuitry and components may be shared by the twomutually exclusive systems to provide a common image processor to reducethe costs and complexity of the vehicle and its electronic features.

For example, and with reference to FIG. 3 , a video mirror 18 a (such asa video mirror of the types described in U.S. Pat. No. 6,690,268, whichis hereby incorporated herein by reference in its entirety) may beprovided, and with a forward facing imager or imaging system 26 includedin a windshield electronics module or in the mirror assembly itself. Theforward facing camera or imager can feed to an EYEQ™ image processingchip or equivalent that is adjacently located in the windshieldelectronics module or interior rearview mirror assembly (oralternatively is located elsewhere in the vehicle, such as at or in aninstrument panel of the vehicle or the like). Thus, when the driver isnormally operating the vehicle and driving forwardly down the road, theforward facing imaging system 26 can be operable to, for example,control headlamps, detect road markings, detect road signs, and/or thelike. However, when the reverse gear of the vehicle is selected at theinitiation of the reversing maneuver and before reversing or reversetravel of the vehicle occurs, the video image from the rear mounted andrearward facing camera 22 can be fed or piped, either digitally or as astandard video signal, and either via a wired or wireless link, to theprocessor to be processed by the EYEQ™ or equivalent image processorcircuitry at or near the front of the vehicle. The video image may alsobe shown, such as on the video mirror (and preferably with graphicoverlays or the like), and optionally with potentially hazardous objectsbeing highlighted and/or enhanced due to the image processing by theEYEQ™ image processor. Similarly, the EYEQ™ image processor, as part ofthe overall control system, may control the visual alerts in any one orall of the rearview mirrors, and preferably may do so with a lowerthreshold sensitivity for such visual alerts as compared to a thresholdsensitivity that may be used with an audible alert. Optionally, theoutputs of any rearward facing non-vision sensors 24 (such as a radarsensor or sensors, an ultrasonic sensor or sensors, and/or an infraredtime-of-flight (TOF) sensor or sensors or the like) may also be fed tothe controller and this can further enhance the accuracy and utility ofthe object detection and the display to and communication with thedriver of the vehicle. Such a system has the commercial advantage thatthe automaker and/or consumer may purchase only one image processor forboth the forward facing imaging (FFI) system or features and the rearfacing imaging (RFI) system or features on the vehicle.

Optionally, a vehicle imaging or vision system may be operable todisplay video images of a scene occurring exteriorly of an equippedvehicle (such as a rearward or forward or sideward scene), such as inresponse to a user input or a driving condition (such as displaying arearward video image in response to the driver shifting a gear actuatorto a reverse gear position), and may be operable to selectively displayother views or otherwise provide an alert in response to an activatingevent such as image processing of image data from one or more cameras atthe vehicle to detect a particular driving condition and/or hazardouscondition. The system thus may display human vision video images on avideo screen (such as a video mirror utilizing aspects of the videodisplays described in U.S. Pat. Nos. 7,490,007; 7,446,650; 7,370,983;7,338,177; 7,274,501; 7,255,451; 7,195,381; 7,184,190; 6,902,284;6,690,268; 6,428,172; 6,420,975; 5,668,663 and/or 5,724,187, which arehereby incorporated herein by reference in their entireties) visible tothe driver of the equipped vehicle when normally operating the equippedvehicle, and may be event triggered to display other images or viewsand/or to provide an alert or warning (such as a visual alert, such as agraphic overlay electronically superimposed with the video images beingdisplayed on the video screen) or the like in response to a detectedevent or driving condition and/or hazardous condition. The system of thepresent invention provides a human vision video display responsive toimage data captured by one or more cameras of the vehicle, while theoutput or outputs of the camera or cameras is, in parallel, beingprocessed by an image processor to machine-vision determine if an objector hazard is within the field of view of the camera and/or if a detectedor determined driving condition is a particular type of drivingcondition, whereby the system may display different images or viewsand/or may generate an alert or warning or the like to the driver of thevehicle. Optionally, in addition to or as an alternate for such a visualalert, an audible and/or haptic alert may be triggered responsive to themachine-vision image processing determining the potential presence of aperson or object that may constitute a hazardous condition when thevehicle is being operated.

For example, and with reference to FIGS. 4-9 , a vehicle imaging orvision system 110 includes a rearward facing camera or imaging sensor112 and a rearward non-imaging sensor 114 (such as a RGB-Z sensor orradar or sonar or ultrasonic sensor or the like) at a rearward portion111 a of a vehicle 111, and a forward facing camera or imaging sensor116 and a forward non-imaging sensor 118 at a forward portion 111 b ofthe vehicle 111. Optionally, the system 110 may include a sidewardcamera or imaging sensor 120 at one or each side 111 c of the vehicle111 (such as at a side exterior rearview mirror assembly 111 d or thelike) and with a generally downward and/or sideward field of view. Animage processor 122 is operable to process image data from each of thecameras 112, 116, 120 and a display device 124 (such as a video displayscreen at an interior rearview mirror assembly 126 of the vehicle or thelike) is operable to display images (such as video images) responsive tothe camera outputs and to the image processor and responsive to asituation or location or event associated with the vehicle as it isdriven by the driver, as discussed below. For example, the cameras maygenerate a video image feed to a graphics engine 125 of or associatedwith the video display screen 124 (and optionally located elsewhere inthe vehicle and remote from the camera and/or the video display screen)for human vision display of the captured images (such as in response toa user input or such as in response to the vehicle actuator being placedin a reverse gear position or the like), while the outputs of thecameras are also in parallel communicated or fed to the image processor122 (located elsewhere in the vehicle, such as at or associated with thevideo display screen or at or associated with a separate ECU, such as aHead Unit ECU controller or a Safety ECU controller or a Chassis ECUcontroller or the like), whereby the image processor processes the imagedata (such as digital image data) to determine a hazardous condition ora driving situation and whereby the displayed image may be adjusted todisplay a different view or video image or an alert or output signal maybe generated in response to such a determination, as also discussedbelow. Alternatively, the image processor may be included at the rearcamera and the desired graphics may be generated at or adjacent the rearcamera 112 itself, and the video feed may include both the images andthe desired graphic data and/or other content.

Forward and rearward and sideward facing cameras 112, 116, 120 maycomprise any suitable camera or imaging sensor, preferably comprising apixelated CCD or CMOS imaging sensor having a plurality of photosensingpixels established on a semiconductor substrate. Preferably, cameras112, 116, 120 are automotive grade color video cameras. Optionally, anddesirably, the cameras may comprise multi-pixel sensors having betterimage resolution than is provided by VGA-type video cameras. Forexample, the camera may comprise a pixelated sensor comprising at leasta 0.5 Megapixel sensor and more preferably at least a 1 Megapixel sensoror a pixelated imaging sensor having more pixels to provide a desired orappropriate resolution of the captured images. Each of the forwardand/or rearward and/or sideward cameras may have a wide angle field ofview (such as shown in FIGS. 5A, 6A and 7 ), and may capture image datarepresentative of a distorted image, whereby image processor 122 mayprocess the captured image data to delineate and correct or account forthe distortion (such as shown in FIG. 5B). As shown in FIGS. 5A and 6A,the wide angle field of view of the forward and rearward cameras 112,116 may extend sidewardly at the front or rear of the vehicle so as toprovide a view of the area in front of or behind the vehicle and towardthe sides of the vehicle. When a particular condition or event isdetected, such as, for example, when it is determined that the vehicleis at a cross-traffic intersection or the like, the video display,responsive to the image processor, may display other views orinformation, such as the sideward directed views of FIGS. 5B and 6B, toassist the driver in seeing approaching traffic at the cross-trafficintersection or the like. Optionally, the forward and/or rearward cameraor cameras may comprise sidewardly facing cameras that capture imagestowards respective sides of the vehicle, and/or may utilize aspects ofthe dual camera imaging systems and/or flip-out cameras described inU.S. Pat. Nos. 6,819,231 and 6,989,736, which are hereby incorporatedherein by reference in their entireties.

The non-imaging sensors may comprise any suitable sensor, such asranging sensors that may determine a distance from the vehicle to adetected object or obstacle at or near or forward/rearward/sideward fromthe vehicle. For example, the non-imaging sensors 114, 118 may comprisean RGB-Z sensor or a radar sensor or a lidar sensor or an infraredsensor (such as an ROD infrared monitoring system or the like) or alaser scanning sensor or a sonar sensor or an ultrasonic sensor or anyother suitable sensor that may operate to enhance the evaluation orprocessing by the system of the area surrounding the equipped vehicle(such as by utilizing aspects of the systems described in U.S. patentapplications, Ser. No. 11/721,406, filed Jun. 11, 2007, now U.S. Pat.No. 8,256,821; and/or Ser. No. 12/266,656, filed Nov. 7, 2008, and/orPCT Application No. PCT/US08/51833, filed Jan. 24, 2008 and publishedOct. 23, 2008 as International Publication No. WO 2008/127752, which arehereby incorporated herein by reference in their entireties). The systemthus may process image data to determine particular driving situationsand may detect objects or the like at or near or approaching thevehicle, while the system may also process or receive outputs of thenon-imaging sensors to further augment the processing of and thedetermination of the driving situations and/or any potential hazardouscondition or the like.

Optionally, the sensor may comprise a RGB-Z sensor. As illustrated inFIG. 10 , RGB-Z combines and synchronizes video imaging withtime-of-flight (TOF) 3D sensing. Such 3D sensing typically uses an arrayof infrared (IR) emitting light emitting diodes (LEDs) or IRfloodlighting using an IR laser diode in conjunction with an array ofIR-sensitive photo sensors, such as is available from CanestaIncorporated, Sunnyvale CA, and such as described in AutomotiveEngineering International (June 2006, pages 34-35), and in U.S. Pat.Nos. 6,323,942 and 6,580,496, which are all hereby incorporated hereinby reference in their entireties. RGB-Z can be used for rear backupsystems and for forward imaging systems, such as for forward parkingsystems and for pedestrian detection and/or the like. Optionally, thepresent invention, preferably in conjunction with RGB-Z, can also beused for side/ground detection such as for the “Japan-view” imagingsystems now common in exterior mirrors used in Japan where a videocamera is located in the exterior mirror assembly at the side of avehicle and viewing generally downwardly to allow the driver of thevehicle to view on an interior-cabin mounted video screen whether thelikes of a child might be present in the blindzone to the side of thevehicle.

The cameras may communicate the captured image data to the graphicsengine 125 and to the shared or common image processor via any suitablemeans. For example, the cameras may wirelessly communicate the capturedimage data to the image processor or may communicate via a wiredconnection or communication link or Ethernet cable or link. For economy,video image transmission via an Ethernet cable can be desirable,particularly when the individual video feeds from multiple video camerasdisposed around the vehicle are being fed to a common image processorand/or electronic control unit and/or video display module or system.Optionally, for example, the connection or link between the imageprocessor and the camera or cameras may be provided via vehicleelectronic or communication systems and the like, and may be connectedvia various protocols or nodes, such as BLUETOOTH®, SCP, UBP, J1850, CANJ2284, Fire Wire 1394, MOST, LIN, FLEXRAY®, Byte Flight and/or the like,or other vehicle-based or in-vehicle communication links or systems(such as WIFI and/or IRDA) and/or the like, depending on the particularapplication of the mirror/accessory system and the vehicle. Optionally,the connections or links may be provided via wireless connectivity orlinks, such as via a wireless communication network or system, such asdescribed in U.S. Pat. No. 7,004,593, which is hereby incorporatedherein by reference in its entirety, without affecting the scope of thepresent invention.

Optionally, the image processor may be disposed at or in or near theinterior rearview mirror assembly of the vehicle, or may be disposedelsewhere in the vehicle. The image processor may comprise any suitableimage processor, such as, for example, an EYEQ2 or an EYEQ1 imageprocessing chip available from Mobileye Vision Technologies Ltd. ofJerusalem, Israel, such as discussed above. Optionally, the imageprocessor may comprise a part of or may be incorporated in a safetyelectronic control unit (ECU) of the vehicle, or a chassis ECU ornavigational ECU or informational ECU or head unit ECU or the like,while remaining within the spirit and scope of the present invention.Optionally, the processor or ECU (such as a head unit ECU of the mirrorassembly or the like) may receive inputs from a navigation system oraudio system or telematics system or the like, and may process suchinputs for the associated controls or features of that system, whilealso processing image data from the cameras for the display feature ofthe imaging system. Preferably, the image processor (and associatedcircuitry, such as memory, network connections, video decoders and/orthe like) may be incorporated into the likes of a Head Unit ECU or thelike in a modular fashion, so as to facilitate inclusion ornon-inclusion by the vehicle manufacturer/Tier 1 supplier of the imageprocessing capability and function of the present invention into theoverall Head Unit ECU or the like.

In the illustrated embodiment, and with reference to FIG. 9 , therearward backup camera 112 (the forward and/or sideward cameras mayfunction in a similar manner) may output a composite NTSC video signal112 a (or alternatively another standard protocol video signal,depending on the particular application of the imaging or visionsystem), which may be communicated to the graphics engine 125 fordisplay of the human vision or non-processed or non-manipulated capturedvideo images at video display 124. Thus, the driver of the vehicle mayview the human vision wide angle video images of the scene occurring atand to the rear of the vehicle, such as in response to a user input orin response to the driver shifting the reverse gear actuator to areverse gear position or the like.

The NTSC video signal 112 a of the backup camera 112 may also becommunicated to a converter 128 that converts the NTSC video signal to adigital video signal, with the digital video signal being input to theimage processor 122. The image processor may process the digital imagedata to determine if an object is present or a hazard exists to the rearof the vehicle (such as by detecting the presence of an object, such asa child or bicycle or the like behind the vehicle). Optionally, theimage processor 122 may also receive an input 114 a from a rangingsensor 114 that is operable to determine a distance from the rear of thevehicle to an object present behind the vehicle. If the image processordetermines that a hazard exists (such as by detecting an object anddetermining that the detected object is within a threshold distance fromthe rear of the vehicle), the image processor may generate an outputsignal that is received by the graphics engine, whereby the videodisplay 124 displays an appropriate image (such as a sideward image orcenter image that encompasses the detected object or such as a graphicoverlay highlighting the detected object or the like) in response to thegraphics engine 125. Optionally, the image processor may generate andcommunicate and output to one or more other systems, such as a warningsystem 130, a braking system 132 and/or a pre-crash system 134 or thelike, in response to a detected hazard and/or driving condition. Theimage processor may also function to detect a driving condition orevent, such as the vehicle approaching and stopping at a cross-trafficintersection or the like, and may generate an output to the graphicsengine 125 so that modified video images (such as sideward views ordifferent video images) are displayed to the driver when a respectiveparticular driving condition or event is detected. Such detection of aparticular event is desirably achieved by the image processor inconjunction with other vehicle functions/sensors that tell the overallsystem whether, for example, the vehicle is stopped or is moving, thevehicle is stopped after having been moving, the vehicle is stoppedfollowing a deceleration typical for a vehicle approaching and stoppingat an intersection or the like during the likes of urban driving, and/orthe like. Optionally, the output of the image processor may be received(such as via a UART data line or the like) and processed by a rangingsensor 136 to determine a distance to the detected object in therearward scene.

Thus, the imaging system may provide a human vision video display(displaying video images of the scene occurring exteriorly of thevehicle as captured by the wide angle rearward facing camera), whilealso providing image processing of captured image data to determine ifan object is present behind the vehicle and/or if a hazard exists duringthe reversing process, and/or if the vehicle is encountering aparticular driving condition or event, whereby the display device orvideo display screen may display an appropriate alert or image orgraphic overlay or modified video image in response to such a detection.The imaging system thus provides a convergence or merging of a humanvision video display and digital image processing to provide enhanceddisplay and/or alert features to a driver of the equipped vehicle. Thus,the video display displays video information specifically tailored forthe particular driving situation of the vehicle, so that the driver isprovided with video information necessary to or desired by the driver ineach driving situation.

The image processor 122 is operable to process image data from eachcamera and may provide an output to the display device that isindicative of various features of the captured images. For example, theimage processor may process the forward camera image data to provide twosideward images (such as shown in FIG. 5B), such as in response to adetermination by the system that the vehicle is at a cross-trafficdriving situation or intersection (such as in response to a detectionthat the vehicle has stopped after driving forward and/or in response toa detection of a traffic sign, such as a stop sign or the like, and anon-moving condition of the vehicle, and/or the like). The system mayoperate to display a desired or appropriate video image (or images) forviewing by the driver of the vehicle as the driver is normally operatingthe vehicle, responsive to an event or situation or driving condition ordetected condition or hazard or the like. Because various drivingconditions or situations are mutually exclusive and because a drivertypically would want to view different areas when in different drivingsituations (such as driving forward, driving rearward, stopping at anintersection or the like), the image processor and system may provide ordisplay the desired or appropriate video images to the driver responsiveto the system determining the driving situation of the vehicle.

For example, if the vehicle is reversing or is about to reverse (such aswhen the driver moves the gear actuator to a reverse gear position), thevideo display may display the wide angle rearward field of view to thedriver of the vehicle, such as in response to the driver placing thereverse gear actuator in the reverse gear position of the vehicle. Aswell as being fed directly to the video screen viewable by the driverexecuting the reversing maneuver, the video feed is also fed in parallelto the image processor (such as the likes of an EYEQ 2 image processoror the like) where frames of the image data being captured aremachine-vision analyzed for characteristics or classificationsindicative or representative of the likes of a child or other person orobject of potential interest or hazard present in the rearward path ofthe vehicle. If an object or hazard is even potentially detectedrearward of the vehicle (such as by such image processing of thecaptured image data to detect an object and/or by determining a distanceto a detected object or the like), the video display (responsive to theimage processor) may display a different field of view (such as a viewthat focuses or enlarges the area at which the object is detected)and/or may highlight the detected object (such as via a color overlay orflashing of the object or the like) and/or the system may otherwiseprovide an alert or warning to the driver of the vehicle and/or thesystem may visually highlight the potentially detected object present inthe video scene being displayed on the video screen to the driver of thevehicle.

Optionally, for example, if the vehicle is being driven in a forwarddirection, the forward facing camera may be operating to capture imagesfor a lane departure warning system (LDW) or the like, with the capturedimage data being processed accordingly (such as by utilizing aspects ofthe systems described in U.S. Pat. Nos. 7,355,524; 7,205,904; 7,038,577;5,929,786 and/or 5,786,772, and/or U.S. patent application Ser. No.11/239,980, filed Sep. 30, 2005, now U.S. Pat. No. 7,881,496; and/orSer. No. 11/315,675, filed Dec. 22, 2005, now U.S. Pat. No. 7,720,580,which are hereby incorporated herein by reference in their entireties).If the vehicle is then stopped, the system, responsive to the changeddriving condition, may determine that the vehicle is stopped at across-traffic situation, such as via image processing to detect a stopsign or the like (such as by utilizing aspects of the systems describedin U.S. Pat. Nos. 5,550,677; 5,670,935; 5,796,094; 5,877,897; 6,313,454;6,353,392 6,396,397; 6,498,620; 7,004,606; 7,038,577 and/or 7,526,103,which are hereby incorporated herein by reference in their entireties)or by determining that the vehicle had been driving in a forwarddirection and then stopped moving. In response to such a determinationthat the vehicle is stopped at a cross-traffic situation, the videodisplay, responsive to the image processor, may display the sidewarddirected views (see FIGS. 5B and 6B) to assist the driver in drivingforward into the intersection or out of a parking space in a parking lotor the like.

Optionally, for example, when a vehicle equipped with the vision systemof the present invention is driving forward along a road, a traffic signrecognition algorithm can recognize a stop sign in the scene ahead ofthe vehicle, and the system may alert the driver to the presence of thestop sign. When the driver stops at the crossroad, both based on imageprocessing analysis of the video image and on other vehicular feeds, thesystem automatically knows that the vehicle is stopped at a crossroad.The forward image processing system of the present invention thusautomatically views and analyzes the video images captured of the leftand right approaching cross traffic, and should the driver commence tomove from the stopped or near-stopped or rolling-stopped condition atthe crossroad, the image processor and system can at minimum visuallyalert the driver via graphic enhancement to draw the attention to thedriver of a left or right approaching vehicle.

Use of a common image processor for a rear reversing event and foranother event, such as stopping at a crossroad or the like, providesmachine-vision processing of the captured image data for a particulardriving condition or event. Given that the particular driving conditions(such as, for example, reversing maneuvers and stopping at a crossroad)can be and are mutually exclusive, a common or shared image processorcan process the received image data in a manner appropriate for thedetected condition, with the common image processor fed with video imagefeeds from a plurality of video imagers/cameras disposed on and aroundthe vehicle (typically with their fields of view external of thevehicle). For example, if the vehicle is reversing or about to reverse,the image processor can process the captured image data captured by therearward facing camera to determine if there is an object of interestrearward of the vehicle and in the rearward path of the vehicle, whileif the vehicle is stopped at a crossroad, the image processor canprocess the sideward views of the forward facing camera to determine ifvehicles are approaching the intersection from the left or right of theequipped vehicle.

Optionally, the vision system may process the captured image data and/ormay be associated with a navigation system to determine the location ofthe vehicle, such as to determine if the vehicle is in an urbanenvironment or rural environment or the like. The navigation system maycomprise any type of navigation system, and may utilize aspects of thesystems described in U.S. Pat. Nos. 6,477,464; 5,924,212; 4,862,594;4,937,945; 5,131,154; 5,255,442; 5,632,092; 7,004,593; 6,678,614;7,167,796 and/or 6,946,978, which are all hereby incorporated herein byreference in their entireties. Optionally, the vehicle speed may bedetermined via processing of the images captured by the imaging sensorsor cameras, such as by utilizing aspects of the systems described inU.S. Pat. No. 7,038,577, which is hereby incorporated herein byreference in its entirety. The system thus may take into account thedriving conditions or geographic location of the vehicle in making thedecision of whether or not to display the sideward views when it isdetermined that the vehicle has stopped at a potential cross-trafficdriving situation.

Optionally, the system may determine that the vehicle is in or atanother driving condition, such as, for example, a parallel parkingsituation. Such a condition may be determined by processing the capturedimage data and detecting the equipped vehicle being driven alongside avacant parking space and being shifted into reverse to back into thevacant parking space. In such a situation, and with reference to FIGS. 7and 8 , the video display 124 may provide an overview 140 of the vehicle(such as an iconistic representation of the vehicle showing thedistances to vehicles or objects forward and rearward of the equippedvehicle, such as in a known manner). The image processor may process thecaptured image data from multiple cameras (such as forward facing camera116, rearward facing camera 112 and opposite sideward facing cameras120) to determine the location of and distance to objects at or near orsurrounding the vehicle. The video display may also provide a main videoview or image 142 that displays to the driver video images of the areaimmediately forward or rearward of the vehicle (such as in response tothe gear shifter or actuator of the vehicle being placed in a forward orreverse gear position). Thus, the main view 142 at the video displayprovides video information specifically necessary to or desired by thedriver during the parking maneuver (or during pulling out of the parkingspace), and optionally may include a graphic overlay, such as a distanceindicator or alert indicator or the like, to further assist the driverduring the vehicle maneuvering situation. Optionally, the bird view oroverview 140 of the vehicle may include border overlays or indicators140 a that are selectively actuated or illuminated or highlighted toindicate to the driver which view the main display 142 is currentlydisplaying to the driver of the vehicle. The video display thus providesa top view of the vehicle while processing image data from theexteriorly directed cameras and sensors to determine and display or flagor highlight locations where the system detects a potential hazard orthe like.

Thus, the vision system of the present invention provides a human visionvideo display of the scene occurring forward and/or rearward and/orsideward of the vehicle to assist the driver during appropriate orrespective driving conditions or situations, such as reversingmaneuvers, lane changes, parking maneuvers and the like. The visionsystem processes captured image data and automatically displays theappropriate video images for the detected particular driving conditionin which the vehicle is being driven. If particular events are detected,such as a cross-traffic situation or a hazardous reversing situation ora parking situation or the like, the system automatically switches thevideo display to display video images or information more suitable tothe driver during the particular detected driving situation. Thus, thevision system of the present invention merges the human vision videodisplay and image processing to provide enhanced information display tothe driver during particular driving conditions. The system thus canknow when the vehicle stops or is otherwise in a particular drivingsituation and can display or flag on the video screen informationpertinent to the driver for the detected particular driving situationand/or can otherwise alert the driver of a potential hazard or the like.The system thus is event triggered and/or hazard detection triggered toprovide an appropriate or desired or necessary view or information tothe driver depending on the particular driving situation and detectedobject and/or hazard at or near or approaching the equipped vehicle.

The system of the present invention may be part of an overall activesafety and sensing system, such as the active safety and sensing systemshown in FIG. 11 . As discussed above, and as illustrated in FIGS. 11and 12 , the system may comprise the combination of machine visionactivity or monitoring (such as for a lane departure warning systemand/or the like) and vehicle control (such as via body/chassis sensorsand sensing). As shown in FIG. 11 , the active safety and sensing systemmay include fusion/combination of outputs from various sensing devices(such as a vision-based or camera-based or image-based sensing systemand a non-image-based sensing system) to provide environmental awarenessat and surrounding the vehicle and may provide partial or completecontrol of the vehicle as it is driven along a road and/or may providealert warnings to the driver of the vehicle of what may be presentenvironmentally exterior of the vehicle and/or what may be hazardousthereat. Machine vision forward facing cameras may be used to providelane departure warning (LDW), traffic sign recognition (TSR), forwardcollision warning (FCW), pedestrian detection, vehicle detection, hazarddetection and/or the like, and these systems may communicate with orcooperate with other systems, such as intelligent headlamp control orautomatic headlamp control (AHC), intelligent light ranging (ILR) (acombination of AHC and ILR may be used for a glide path automaticheadlamp control, for example, where the headlamps are actively ordynamically adjusted so that the beam pattern forward of the equippedvehicle can be configured to illuminate the road just ahead of anapproaching vehicle), lane keep assist (LKA) (where the steering wheelmay variably provide resistance to turning to further alert the driverof a detected potentially hazardous condition and/or may actively turnor control the steering system of the vehicle so as to mitigate or avoidan imminent potential collision) and/or the like.

Much of the active sensing system builds on the existing alphabet of thevision system foundation that comprises camera-based headlamp control,camera-based lane detection, camera-based sign detection/recognition,camera-based object detection. Optionally, an LDW system or function maybe extended to an LKA system or function by tracking the lane alongwhich the vehicle is driven and controlling the steering torque to aidthe driver in maintaining the vehicle in the lane, such as shown in FIG.14 . Optionally, the system may include a map input or geographicallocation input (such as from an onboard or an external GPS-basednavigational system), whereby the vehicle safety system may begeographically/locally customized to operate differently or may processthe image data differently or the like, in response to the map input orgeographical location input indicative of the particular geographicallocation of the equipped vehicle at that moment in time. Optionally, andpreferably, the map data/GPS derived information relating to, forexample, the curvature and/or bank angle of a highway exit or entranceramp may tie into the automatic headlamp control and/or the directioncontrol of the headlamp beam.

Optionally, for example, map data (such as longitude/latitude/altitudecoordinates) may be provided in connection with or fused with a TSRsystem and/or an AHC/ILR system and/or an automatic cruise control (ACC)system to enhance performance of the TSR system and/or the AHC/ILRsystem and/or the ACC system. For example, the control system may fuse atraffic sign recognition (TSR) function or feature with map data toachieve better speed limit recognition performance. The system mayimprove or enhance performance such that TSR can be used for automaticACC target speed setting, ideally in all driving scenarios, such as oncontrolled access freeways. The system may have defined rules to handleinconsistent information between map data and video image data, and mayhave a defined interface to the ACC to allow the ACC to receive targetspeed from or responsive to the video camera. Optionally, for example,an ACC lane assignment may be based on camera lane information, and animproved or enhanced ACC target vehicle lane assignment may be achievedby using front camera lane prediction and/or map data to correct theradar sensor's lane model. Such an approach may make the ACC moreaccurate and dynamic. The system design and algorithm thus may definethe lane curvature interface between the forward facing camera imagedata and the radar output. Optionally, and with reference to FIG. 15 ,an AHC/ILR system may be fused with map data to enhance adjustment ofthe vehicle lights. For example, an AHC/ILR may utilize knowledge ofroad curvature and slope, and may, for example, predict thatdisappearing lights of another vehicle are due to the leading vehicleentering a valley or different elevation, whereby the system mayrecognize that the disappearing lights are likely to reappear shortly.Such AHC/ILR and map data fusion may improve or enhance the AHC/ILRperformance beyond the normal capabilities of the forward facing cameraand image processor. The system may have defined rules and algorithms asto how to interpret the map data and how to adjust the light control.Optionally, the vehicle may be provided with advanced driver assistancesystems (ADAS) map data or the like. The map data or other data orinformation may be provided to or acquired by the system via anysuitable means, such as via software or hardware of the system or viadownloading map data or other information or data, such as real timedownloading of map data or the like, such as Google Maps map data or thelike. Optionally, the system may receive inputs from a Car2Cartelematics communication system or a Car2X telematics communicationsystem or the like.

Optionally, camera data or information may be fused with radar data orinformation (or with other non-vision based data, such as fromultrasonic sensors or infrared sensors or the like) to derive objectinformation and emergency braking may be initiated in response to suchobject detection. For example, camera and radar information may be fusedto derive object information sufficiently accurate to initiate emergencybraking. Target information may be merged between radar and cameraoutputs and any potential synchronicity challenges may be addressed.Optionally, an LKA system and emergency braking system may cooperate toprovide semi-autonomous driving, such as by allowing lateral andlongitudinal control over the vehicle while the driver is the defaultredundant vehicle control mechanism with 0 ms latency to assume control(LKA, emergency braking). Such a system may reduce or mitigate problemsassociated with the driver becoming distracted, and the system maycontrol the vehicle with some latency. It is envisioned that such acontrol system may allow the driver to read a book or type an emailwhile the car is driving itself, such as on a controlled access freewayor the like. The system may include improved fusion algorithms toinclude side and/or rear facing sensors, and may include suitabledecision algorithms that facilitate autonomous vehicle control.

Optionally, as discussed above, the image processor of the system maycomprise an advanced image processing platform, such as, for example, anEYEQX image processing chip, such as an EYEQ2 or an EYEQ1 imageprocessing chip available from Mobileye Vision Technologies Ltd. ofJerusalem, Israel, such as discussed above, and such as shown anddescribed in FIGS. 16, 16A and 16B. Optionally, the advanced imageprocessing platform may provide a new hardware platform to addresschallenges:

-   -   Address higher Resolution Imagers (such as, for example, a        MI-1000 or the like);    -   Address a two-box design (for design flexibility, safety        requirements, and/or the like);    -   Address scalability: define a vehicle video bus to add        additional cameras without having to change environmental        awareness platform;    -   Support new vehicle interfaces, such as, for example, Flexray or        the like;    -   Solve need for synchronicity and fusion;    -   Resolve inefficient use of resources between EYEQ2 and S12X (for        example, lots of RAM in EYEQ2 may not be accessible to S12X),        may eliminate S12X;    -   Address scalability need, for example, EYEQX may be a platform        including at least an EYEQX low and an EYEQX high; and/or    -   Address need for Autosar compliance and ISO 26262 safety design        consequences.

Optionally, on an EYEQX platform, the system may allow writing of codedirectly on EYEQ (for example, the system may compile, link and loadcode in EYEQ without need for Mobileye). Optionally, the system maydefine APIs between system code and Mobileye code on the same processor.Optionally, the vehicle interfaces may be negotiated between theMobileye and the system. Optionally, they system may include a ST 16-bitmicro core added to the EYEQX. Optionally, a Port S12X code may beincorporated onto the EYEQX.

For example, and with reference to FIGS. 16, 16A and 16B, an activesafety and sensing and alert system of the present invention may includea control module that includes an environmental awareness control modulethat is in communication with the sensors and other control modules orsystems of the active safety and sensing and alert system and/or of thevehicle, and may include a central processor operable to perform imageprocessing and/or other processing. The system includes a forward facingcamera/imager that is in communication with and/or is in connection withthe control module or processor, such as via a video bus of the vehicle,such as MOST or the like, or via an Ethernet link that follows anEthernet protocol or via the likes of a twin wire wired connectioncarrying the likes of NTSC video signals, and one or more other sensors(such as a front radar sensor and right and left side radar sensors orthe like, such as for side object detection functions and/or lane changeassist functions) in communication with the control module via one ormore sensor buses or the like. Optionally, the system may includeadditional cameras or image sensors, such as a rearward facing cameraand left and right cameras and optionally another forward vision cameraand an RGB-Z sensor, which are in communication with the control modulevia a second video bus of the vehicle. The control module may processthe image data and a video display may display video images responsiveto the control module. Optionally, the system may include other sensors,such as smart sensors and/or park sensors and/or the like, which may bein communication with the control module via a LIN network of thevehicle or the like.

The control module may include or be operable to perform variousfunctions or algorithms, such as, for example, warning algorithms,fusion algorithms or vehicle control algorithms, or may process orprovide map data or AD ASIS reconstructor, network management, or videooverlay generation and/or the like. The control module may communicatewith one or more vehicle control modules, such as, for example, asupervisory control module, a brake control module, a chassis controlmodule, a steering control module, or an engine control module and/orthe like, such as via a chassis bus of the vehicle, whereby the controlmodule may control one or more of the vehicle control modules inresponse to the image processing or in response to processing of othersensors at the vehicle or in response to other inputs to the controlmodule. The control module may also be in communication with a radioand/or navigation system of the vehicle or a body control module of thevehicle, such as via a body bus of the vehicle, and may, for example,control the speakers of the vehicle to provide an audible warning, suchas in response to the image processing or processing of other sensors atthe vehicle or other inputs to the control module. Thus, the presentinvention may provide a single control module and image processor thatis operable to process image data and other data or signals receivedfrom multiple sensors disposed at the vehicle and may control a videodisplay and/or other control systems of the vehicle in response to suchprocessing and/or in response to other inputs to the control module.

Optionally, the camera or image device or cameras or image devices maycomprise a “one-box” design with the imager and associated imageprocessing circuitry closely packaged together in a unit or box, or thecamera or image device may comprise a “two-box” design with the imagerconnected via a wire connection to a second “box” that includes theimage processing circuitry, such as by a cable or by a network bus orthe like. A “one-box” design, such as with all processing located in thecamera, may not be sustainable when adding more features and makingsafety relevant decisions. The system may utilize any suitable videotransfer alternatives (such as, for example, MOST, Firewire, LVDS, USB2.0, Ethernet and/or the like) with/without lossless compression. Thesystem may include two box communication architecture. Optionally, indeveloping such a system, a data acquisition hardware and softwaresystem may allow for fusion system development, where the system mayrecord multiple sensor data streams (such as from cameras, radar, mapdata and/or the like) in real time and may be able to synchronously playback for hardware in the loop testing.

Optionally, and with reference to FIG. 17 , a driver active safety (DAS)control or control module 310 may include an image processor 312, avision core 314 and a fusion core 316 co-established on and coplanar ona common substrate 318 (such as a common semiconductor wafer or die,such as a silicon semiconductor wafer or die) to provide a central DAScontrol unit or module for the vehicle. The image processor 312processes image data received from one or more imaging sensors 320 (suchas received from a forward facing camera or imaging sensor and/or asideward or rearward facing camera or imaging sensor), such as forautomatic headlamp control, lane departure warning, traffic signrecognition and/or the like. The vision core 314 may receive data/inputfrom the image processor 312 and may receive image data from one or froma plurality of vision cameras 322 and may manipulate the visual imagesfor displaying the desired or appropriate captured images for viewing bythe driver of the vehicle (such as on a video display screen 324 that isresponsive to the control module). The common establishment of an imageprocessor and a vision core coplanar on the same or common substrate,and both established at least partially in a common wafer processingstep or series of steps, enables economic and packaging friendlyminiaturization of the features provided, and minimizes the need and useof ancillary external electrical components, connectors and cabling.

The vision core 314 may process the images (and may superimpose uponthem alert or icons or similar graphic overlays based on parallel imageprocessing of the captured image by the image processor) to provide, forexample, a panoramic view around the vehicle for viewing by the driverof the vehicle or to provide the likes of a bird's eye view or asurround vision view or an entering traffic cross view or the like forviewing by the driver of the vehicle. The display 324 may comprise anysuitable display, such as a video display screen or a backlit liquidcrystal display screen or a reconfigurable video display screen or thelike. The display may be disposed at the interior rearview mirror andmay be viewable through the reflective element (such as a transflectivereflective element that is partially transmissive of light therethroughand partially reflective of light incident thereon) by a driver viewingthe mirror when the mirror is normally mounted in a vehicle.

The fusion core 316 of the control module 310 may receive inputs fromvarious systems (such as systems of the vehicle or remote from thevehicle), such as a GPS and/or navigational system 326, a Car2Car orCar2X telematics system 328, an infotainment system 330 or the like. Thefusion core 316 may also receive inputs from other vehicle or systemsensors, such as non-imaging sensors 332, such as radar sensors orinfrared sensors or IROD sensors or TOF sensors or ultrasonic sensors orthe like, and/or may receive information or data on vehicle status fromvarious vehicle systems or devices 334, such as vehicle speed, vehiclesteering wheel angle, yaw rate of the vehicle, type of vehicle,acceleration of the vehicle and/or the like. Optionally, outputs anddata from the non-imaging sensors 332 may be received directly by or atthe image processor 312 to enhance object detection (useful, forexample, for forward collision warning or pedestrian detection or thelike) or headlamp control or lane departure warning or the like. Thefusion core 316 may receive input and data from the image processor 312and may fuse this information with input or inputs or data from one ormore of the other systems or sensors to enhance the processing and/ordecision making and/or control of the control system. Some informationmay be communicated from the image processor to the fusion core and someinformation may be communicated from the fusion core to the imageprocessor, depending on the particular function being performed by thecontrol module. Optionally, the fusion core may be incorporated integralto or directly into the construction and architecture of the imageprocessor or the vision core. Optionally, for applications where sensorfusion is not of particular utilization or importance, an imageprocessor and a vision core may be co-established on the commonsubstrate in accordance with the present invention.

As shown in FIG. 17 , the control module 310 may provide an output toone or more vehicle controls 336, such as to a vehicle body controlmodule, a vehicle chassis control module, a vehicle accessory controlmodule or other vehicle function control (such as a supervisory controlmodule, a brake control module, a steering control module, or an enginecontrol module) and/or the like, such as via a network bus of thevehicle, such as a safety CAN bus of the vehicle. The control module maycontrol one or more of the vehicle control modules or functions inresponse to the image processing and/or in response to processing ofother sensors at the vehicle or in response to other inputs to thecontrol module, to achieve the likes of pre-crash warning/mitigation,automatic braking, automatic seat belt tensioning and/or the like.

Optionally, the control module 310 may include an application core 338for hosting other system programs or algorithms or software. Forexample, this may provide a processor and/or associated memory foroperating a variety of software packages that may be particular to agiven automaker's or driver's preferences, desires and/or needs.Preferably, a firewall is provided to segregate and separate and protectsuch applications running on the application core. For example, videoand/or text messages may be recorded to and played back from such anapplication core. For example, and using the likes of a telematicssystem (such as ONSTAR® or the like), a tour guide function may bedownloaded to the application core such that when the driver is drivingthrough, for example, a historic district or nature preserve or the like(or other place of interest), information may be conveyed to the driver(such as audibly or visually) that is particular to a house or locationor landmark that is being then passed by the equipped vehicle. In thisregard, the map data may be fused or accessed to assist in determiningthe location or any such houses or landmarks and/or the relativelocation of the equipped vehicle.

Thus, the control module of the present invention provides an imageprocessor and vision core on a single circuit chip, and commonlyestablished on the chip and coplanar with one another. The controlmodule thus has a single chip or substrate (such as a silicon substrate)with an image processor and vision core established thereat. Theconnection to the various sensors and systems and devices of the vehiclemay be made via any suitable connecting or communicating means, such as,for example, via a wireless communication link or a wired link, anEthernet link operating under an Ethernet protocol, and may utilize anysuitable communication protocol, such as BLUETOOTH®, SCP, UBP, J1850,CAN J2284, Fire Wire 1394, MOST, LIN, FLEXRAY®, Byte Flight, Autosarand/or the like.

Optionally, and as shown in FIG. 18 , an imaging device 410 suitable foruse with the alert system and/or active safety and sensing system of thepresent invention (or for other uses in an overall active safety systemof a vehicle) may include a semiconductor substrate 412 (preferably asilicon substrate) with a night vision imaging array 414 configured tobe principally sensitive to near infrared radiation and a machine visionimaging array 416 configured to be principally sensitive to visiblelight, commonly established (such as by CMOS processing) on a commonsemiconductor substrate or die. The night vision imaging array 414 isconfigured to be principally sensitive to near infrared (IR) light (andthus may be provided with an IR or near IR pass filter that principallypasses near infrared light and that mostly or wholly rejects visiblelight), while the machine vision imaging array 416 may have a spectralfilter(s) 416 a, such as an IR reject filter that limits orsubstantially precludes the sensor from being flooded by IR radiationand/or a spectrally selective filter (such as a RGB filter or ared/clear filter or the like) that selectively transmits the likes ofred visible light to assist machine vision recognition anddiscrimination of the likes of headlamps and taillights and stop signsand/or the like.

The two distinct imagers (a night vision imaging array that typically issensitive to IR light and has a IR pass filter to reject and not besensitive to and saturated by visible light and a machine vision imagingarray that has an IR reject filter and may have an RGB filter or ared/clear filter to provide color discrimination) are disposed on thesame or common substrate 412 or may be disposed in front of or at thecommon substrate, and each imager may have a respective lens 414 a, 416b disposed thereat. An image processor 418 is also disposed on or at thesame or common substrate 412 (and may be created or established thereatsuch as by CMOS processing in the same chip manufacturing process as isthe imaging arrays), along with miscellaneous circuitry 420, such asmemory, an A/D converter, a D/A converter, CAN controllers and/or thelike. The imaging device thus has two imaging arrays and/or an imageprocessor and/or ancillary miscellaneous circuitry on or at the samesubstrate or chip, so that a single chip may provide imaging with twodistinct imagers and/or a single chip may provide image processingcircuitry on the same substrate or chip as one or more imagers orimaging arrays, so as to provide vision capabilities and/or imageprocessing on one substrate instead of having two separate imagingdevices.

The video display screen device or module may comprise any suitable typeof video screen and is operable to display images in response to aninput or signal from a control or imaging system. For example, the videodisplay screen may comprise a multi-pixel liquid crystal module (LCM) orliquid crystal display (LCD), preferably a thin film transistor (TFT)multi-pixel liquid crystal display (such as discussed below), or thescreen may comprise a multi-pixel organic electroluminescent display ora multi-pixel light emitting diode (LED), such as an organic lightemitting diode (OLED) or inorganic light emitting diode display or thelike, or a passive reflective and/or backlit pixelated display, or anelectroluminescent (EL) display, or a vacuum fluorescent (VF) display orthe like. For example, the video display screen may comprise a videoscreen of the types disclosed in U.S. Pat. Nos. 7,446,650; 7,370,983;7,338,177; 7,274,501; 7,255,451; 7,195,381; 7,184,190; 7,012,727;6,902,284; 6,690,268; 6,428,172; 6,420,975; 5,668,663 and/or 5,724,187,and/or U.S. patent application Ser. No. 12/414,190, filed Mar. 30, 2009,now U.S. Pat. No. 8,154,418; Ser. No. 10/538,724, filed Jun. 13, 2005and published Mar. 9, 2006 as U.S. Patent Publication No. US2006/0050018; Ser. No. 11/226,628, filed Sep. 14, 2005 and publishedMar. 23, 2006 as U.S. Publication No. US 2006-0061008; Ser. No.12/091,525, filed Apr. 25, 2008, now U.S. Pat. No. 7,855,755; Ser. No.09/585,379, filed Jun. 1, 2000; and/or Ser. No. 12/578,732, filed Oct.14, 2009 and published Apr. 22, 2010 as U.S. Publication No.US-2010-0097469, which are hereby incorporated herein by reference intheir entireties. Optionally, video displays may be disposed at therearview mirror assemblies and may be operable to display video imagesof the rearward scene, such as by utilizing aspects of the displaysdescribed in U.S. patent application Ser. No. 11/933,697, filed Nov. 1,2007, now U.S. Pat. No. 7,777,611, which is hereby incorporated hereinby reference in its entirety. Each mirror thus may provide a videodisplay (such as including a video display screen disposed behind andviewable through a transflector or transflective mirror reflector of areflective element) and the display may be larger if provided as adisplay-on-demand type of display behind a transflective mirrorreflector of the reflective element and viewable through thetransflective mirror reflector of the reflective element.

Optionally, the video display module may provide a graphic overlay toenhance the driver's cognitive awareness of the distances to objects tothe rear of the vehicle (such as by utilizing aspects of the systemsdescribed in U.S. Pat. Nos. 5,670,935; 5,949,331; 6,222,447 and6,611,202; and/or PCT Application No. PCT/US08/76022, filed Sep. 11,2008, which are hereby incorporated herein by reference in theirentireties. Such graphic overlays may be generated at or by the cameracircuitry or the mirror or display circuitry. Optionally, the displaymodule may comprise a high luminance 3.5 inch video display or a 4.3inch video display, preferably having a display intensity of at leastabout 400 candelas per square meter (cd/m²) as viewed through thereflective element (preferably as viewed through a transflective mirrorreflector of the transflective reflective element) by a person viewingthe mirror reflective element, more preferably at least about 1000 cd/m²as viewed through the reflective element (preferably as viewed through atransflective mirror reflector of the transflective reflective element)by a person viewing the mirror reflective element, and more preferablyat least about 1500 cd/m² as viewed through the reflective element(preferably as viewed through a transflective mirror reflector of thetransflective reflective element) by a person viewing the mirrorreflective element.

The imaging device and control and image processor may comprise anysuitable components, and may utilize aspects of the cameras and visionsystems described in U.S. Pat. Nos. 5,550,677; 5,877,897; 6,498,620;5,670,935; 5,796,094; 6,396,397; 6,806,452; 6,690,268; 7,005,974;7,123,168; 7,004,606; 6,946,978; 7,038,577; 6,353,392; 6,320,176;6,313,454 and 6,824,281, and/or U.S. patent application Ser. No.12/508,840, filed Jul. 24, 2009 and published Jan. 28, 2010 as U.S.Publication No. US-2010-0020170, and/or U.S. provisional applications,Ser. No. 61/303,054, filed Feb. 10, 2010; Ser. No. 61/785,565, filed May15, 2009; Ser. No. 61/186,573, filed Jun. 12, 2009; and/or Ser. No.61/238,862, filed Sep. 1, 2009, which are all hereby incorporated hereinby reference in their entireties. Optionally, the circuit board or chipmay include circuitry for the imaging array sensor and or otherelectronic accessories or features, such as by utilizingcompass-on-a-chip or EC driver-on-a-chip technology and aspects such asdescribed in U.S. Pat. No. 7,255,451 and/or 7, 480,149, and/or U.S.patent application Ser. No. 11/226,628, filed Sep. 14, 2005 andpublished Mar. 23, 2006 as U.S. Publication No. US 2006-0061008, whichare hereby incorporated herein by reference in their entireties. Thecamera or camera module may comprise any suitable camera or imagingsensor, and may utilize aspects of the cameras or sensors described inU.S. Pat. No. 7,480,149 and/or U.S. patent application Ser. No.12/091,359, filed Apr. 24, 2008 and published Oct. 1, 2009 as U.S.Publication No. US 2009-0244361; and/or Ser. No. 10/534,632, filed May11, 2005 and published Aug. 3, 2006 as U.S. Patent Publication No.US-2006-0171704, now U.S. Pat. No. 7,965,336, and/or U.S. provisionalapplication Ser. No. 61/303,054, filed Feb. 10, 2010, which are allhereby incorporated herein by reference in their entireties. The imagingarray sensor may comprise any suitable sensor, and may utilize variousimaging sensors or imaging array sensors or cameras or the like, such asa CMOS imaging array sensor, a CCD sensor or other sensors or the like,such as the types described in U.S. Pat. Nos. 5,550,677; 5,760,962;5,715,093; 5,877,897; 6,922,292; 6,757,109; 6,717,610; 6,590,719;6,201,642; 6,498,620; 5,796,094; 6,097,023; 6,320,176; 6,559,435;6,831,261; 6,806,452; 6,396,397; 6,822,563; 6,946,978; 7,339,149;7,038,577 and 7,004,606; and/or U.S. patent application Ser. No.11/315,675, filed Dec. 22, 2005, now U.S. Pat. No. 7,720,580; and/or PCTApplication No. PCT/US2003/036177 filed Nov. 14, 2003, and publishedJun. 3, 2004 as PCT Publication No. WO 2004/047421, and/or PCTApplication No. PCT/US2008/076022, filed Sep. 11, 2008 and publishedMar. 19, 2009 as International Publication No. WO 2009/036176, and/orPCT Application No. PCT/US2008/078700, filed Oct. 3, 2008 and publishedApr. 9, 2009 as International Publication No. WO 2009/046268, which areall hereby incorporated herein by reference in their entireties.

The camera module and circuit chip or board and imaging sensor of thepresent invention may be implemented and operated in connection withvarious vehicular vision-based systems, and/or may be operable utilizingthe principles of such other vehicular systems, such as a vehicleheadlamp control system, such as the type disclosed in U.S. Pat. Nos.5,796,094; 6,097,023; 6,320,176; 6,559,435; 6,831,261; 7,004,606 and7,339,149, and U.S. patent application Ser. No. 11/105,757, filed Apr.14, 2005, now U.S. Pat. No. 7,526,103, and U.S. provisional applicationSer. No. 61/785,565, filed May 15, 2009, which are all herebyincorporated herein by reference in their entireties, a rain sensor,such as the types disclosed in commonly assigned U.S. Pat. Nos.6,353,392; 6,313,454 and/or 6,320,176, and/or U.S. patent applicationSer. No. 11/201,661, filed Aug. 11, 2005, now U.S. Pat. No. 7,480,149,which are hereby incorporated herein by reference in their entireties, avehicle vision system, such as a forwardly, sidewardly or rearwardlydirected vehicle vision system utilizing principles disclosed in U.S.Pat. Nos. 5,550,677; 5,760,962; 5,877,897; 5,949,331; 6,222,447;6,302,545; 6,396,397; 6,498,620; 6,523,964; 6,611,202; 6,201,642;6,690,268; 6,717,610; 6,757,109; 6,802,617; 6,806,452; 6,822,563;6,891,563; 6,946,978; 7,205,904 and 7,355,524, and/or in U.S. patentapplication Ser. No. 10/643,602, filed Aug. 19, 2003, now U.S. Pat. No.7,859,565, which are all hereby incorporated herein by reference intheir entireties, a trailer hitching aid or tow check system, such asthe type disclosed in U.S. Pat. No. 7,005,974, which is herebyincorporated herein by reference in its entirety, a reverse or sidewardimaging system, such as for a lane change assistance system or lanedeparture warning system or for a blind spot or object detection system,such as imaging or detection systems of the types disclosed in U.S. Pat.Nos. 7,355,524; 7,205,904; 7,038,577; 5,929,786 and/or 5,786,772, and/orU.S. patent application Ser. No. 11/239,980, filed Sep. 30, 2005, nowU.S. Pat. No. 7,881,496; and/or Ser. No. 11/315,675, filed Dec. 22,2005, now U.S. Pat. No. 7,720,580, and/or U.S. provisional applications,Ser. No. 60/628,709, filed Nov. 17, 2004; Ser. No. 60/614,644, filedSep. 30, 2004; Ser. No. 60/618,686, filed Oct. 14, 2004; Ser. No.60/638,687, filed Dec. 23, 2004; and/or Ser. No. 61/238,862, filed Sep.1, 2009, which are hereby incorporated herein by reference in theirentireties, a video device for internal cabin surveillance and/or videotelephone function, such as disclosed in U.S. Pat. Nos. 5,877,897;6,690,268 and/or 7,370,983, and/or PCT Application No. PCT/US03/40611,filed Dec. 19, 2003 and published Jul. 15, 2004 as PCT Publication No.WO 2004/058540, and/or U.S. patent application Ser. No. 10/538,724,filed Jun. 13, 2005 and published Mar. 9, 2006 as U.S. Publication No.US-2006-0050018, and/or U.S. provisional applications, Ser. No.60/630,061, filed Nov. 22, 2004; and Ser. No. 60/667,048, filed Mar. 31,2005, which are hereby incorporated herein by reference in theirentireties, a traffic sign recognition system, a system for determininga distance to a leading or trailing vehicle or object, such as a systemutilizing the principles disclosed in U.S. Pat. No. 6,396,397 and/or7,123,168, which are hereby incorporated herein by reference in theirentireties, and/or the like.

Optionally, the circuit board or chip may include circuitry for theimaging array sensor and or other electronic accessories or features,such as by utilizing compass-on-a-chip or EC driver-on-a-chip technologyand aspects such as described in U.S. Pat. No. 7,255,451 and/or U.S.patent application Ser. No. 11/201,661, filed Aug. 11, 2005, now U.S.Pat. No. 7,480,149; and/or Ser. No. 11/226,628, filed Sep. 14, 2005 andpublished Mar. 23, 2006 as U.S. Publication No. US 2006-0061008, whichare hereby incorporated herein by reference in their entireties.

Optionally, the interior and/or exterior mirror assemblies may comprisean electro-optic or electrochromic mirror assembly and may include anelectro-optic or electrochromic reflective element. The electrochromicmirror element of the electrochromic mirror assembly may utilize theprinciples disclosed in commonly assigned U.S. Pat. Nos. 6,690,268;5,140,455; 5,151,816; 6,178,034; 6,154,306; 6,002,544; 5,525,264;5,610,756; 5,406,414; 5,253,109; 5,076,673; 5,073,012; 5,117,346;5,668,663; 5,910,854; 5,142,407 and/or 4,712,879, which are herebyincorporated herein by reference in their entireties, and/or asdisclosed in the following publications: N. R. Lynam, “ElectrochromicAutomotive Day/Night Mirrors”, SAE Technical Paper Series 870636 (1987);N. R. Lynam, “Smart Windows for Automobiles”, SAE Technical Paper Series900419 (1990); N. R. Lynam and A. Agrawal, “Automotive Applications ofChromogenic Materials”, Large Area Chromogenics: Materials and Devicesfor Transmittance Control, C. M. Lampert and C. G. Granquist, EDS.,Optical Engineering Press, Wash. (1990), which are hereby incorporatedby reference herein in their entireties; and/or as described in U.S.Pat. No. 7,195,381, which is hereby incorporated herein by reference inits entirety. Optionally, the electrochromic circuitry and/or a glaresensor (such as a rearward facing glare sensor that receives light fromrearward of the mirror assembly and vehicle through a port or openingalong the casing and/or bezel portion and/or reflective element of themirror assembly) and circuitry and/or an ambient light sensor andcircuitry may be provided on one or more circuit boards of the mirrorassembly. The mirror assembly may include one or more other displays,such as the types disclosed in U.S. Pat. No. 5,530,240 and/or 6,329,925,which are hereby incorporated herein by reference in their entireties,and/or display-on-demand transflective type displays, such as the typesdisclosed in U.S. Pat. Nos. 7,274,501; 7,255,451; 7,195,381; 7,184,190;5,724,187 and/or 6,690,268, and/or in U.S. patent application Ser. No.11/226,628, filed Sep. 14, 2005 and published Mar. 23, 2006 as U.S.Publication No. US 2006-0061008; Ser. No. 10/538,724, filed Jun. 13,2005 and published Mar. 9, 2006 as U.S. Patent Publication No. US2006/0050018; and/or Ser. No. 11/912,576, filed Oct. 25, 2007, now U.S.Pat. No. 7,626,749, and/or PCT Application No. PCT/US03/29776, filedSep. 9, 2003 and published Apr. 1, 2004 as International Publication No.WO 2004/026633, which are all hereby incorporated herein by reference intheir entireties. The thicknesses and materials of the coatings on thesubstrates, such as on the third surface of the reflective elementassembly, may be selected to provide a desired color or tint to themirror reflective element, such as a blue colored reflector, such as isknown in the art and such as described in U.S. Pat. Nos. 5,910,854;6,420,036 and/or 7,274,501, and in PCT Application No. PCT/US03/29776,filed Sep. 9, 2003 and published Apr. 1, 2004 as InternationalPublication No. WO 2004/026633, which are all hereby incorporated hereinby reference in their entireties.

Optionally, the interior rearview mirror assembly may comprise aprismatic mirror assembly or a non-electro-optic mirror assembly or anelectro-optic or electrochromic mirror assembly. For example, theinterior rearview mirror assembly may comprise a prismatic mirrorassembly, such as the types described in U.S. Pat. Nos. 7,249,860;6,318,870; 6,598,980; 5,327,288; 4,948,242; 4,826,289; 4,436,371 and4,435,042; and PCT Application No. PCT/US2004/015424, filed May 18,2004, and published on Dec. 2, 2004, as International Publication No. WO2004/103772, which are hereby incorporated herein by reference in theirentireties. Optionally, the prismatic reflective element may comprise aconventional prismatic reflective element or prism or may comprise aprismatic reflective element of the types described in U.S. Pat. Nos.7,420,756; 7,274,501; 7,249,860; 7,338,177 and/or 7,255,451, and/or PCTApplication No. PCT/US03/29776, filed Sep. 19, 2003, and published Apr.1, 2004 as International Publication No. WO 2004/026633; and/or PCTApplication No. PCT/US2004/015424, filed May 18, 2004, and published onDec. 2, 2004, as International Publication No. WO 2004/103772; and U.S.provisional application, Ser. No. 60/525,952, filed Nov. 26, 2003, whichare all hereby incorporated herein by reference in their entireties,without affecting the scope of the present invention. A variety ofmirror accessories and constructions are known in the art, such as thosedisclosed in U.S. Pat. Nos. 5,555,136; 5,582,383; 5,680,263; 5,984,482;6,227,675; 6,229,319 and 6,315,421 (the entire disclosures of which arehereby incorporated by reference herein), that can benefit from thepresent invention.

Optionally, the mirror assembly and/or reflective element may includeone or more displays, such as for the accessories or circuitry describedherein. The displays may be similar to those described above, or may beof types disclosed in U.S. Pat. No. 5,530,240 and/or 6,329,925, whichare hereby incorporated herein by reference in their entireties, and/ormay be display-on-demand or transflective type displays, such as thetypes disclosed in U.S. Pat. Nos. 7,195,381; 6,690,298; 5,668,663 and/or5,724,187, and/or in U.S. patent application Ser. No. 11/226,628, filedSep. 14, 2005 and published Mar. 23, 2006 as U.S. Publication No.US-2006-0061008; and/or Ser. No. 10/993,302, filed Nov. 19, 2004, nowU.S. Pat. No. 7,338,177; and/or in U.S. provisional applications, Ser.No. 60/525,952, filed Nov. 26, 2003; Ser. No. 60/717,093, filed Sep. 14,2005; and/or Ser. No. 60/732,245, filed Nov. 1, 2005, and/or in PCTApplication No. PCT/US03/29776, filed Sep. 19, 2003, and published Apr.1, 2004 as International Publication No. WO 2004/026633, which are allhereby incorporated herein by reference in their entireties. Optionally,a prismatic reflective element may comprise a display on demand ortransflective prismatic element (such as described in PCT ApplicationNo. PCT/US03/29776, filed Sep. 19, 2003, and published Apr. 1, 2004 asInternational Publication No. WO 2004/026633; and/or U.S. patentapplication Ser. No. 10/993,302, filed Nov. 19, 2004, now U.S. Pat. No.7,338,177; and/or U.S. provisional application, Ser. No. 60/525,952,filed Nov. 26, 2003, which are all hereby incorporated herein byreference in their entireties) so that the displays are viewable throughthe reflective element, while the display area still functions tosubstantially reflect light, in order to provide a generally uniformprismatic reflective element even in the areas that have displayelements positioned behind the reflective element.

Optionally, the display and any associated user inputs may be associatedwith various accessories or systems, such as, for example, a tirepressure monitoring system or a passenger air bag status or a garagedoor opening system or a telematics system or any other accessory orsystem of the mirror assembly or of the vehicle or of an accessorymodule or console of the vehicle, such as an accessory module or consoleof the types described in U.S. Pat. Nos. 6,877,888; 6,824,281;6,690,268; 6,672,744; 6,386,742 and 6,124,886, and/or, and/or PCTApplication No. PCT/US03/03012, filed Jan. 31, 2003, and published Aug.7, 2003 as International Publication No. WO 03/065084, and/or PCTApplication No. PCT/US03/40611, filed Dec. 19, 2003, and published Jul.15, 2004 as International Publication No. WO 2004/058540, and/or PCTApplication No. PCT/US04/15424, filed May 18, 2004, and published onDec. 2, 2004, as International Publication No. WO 2004/103772, which arehereby incorporated herein by reference in their entireties.

Changes and modifications to the specifically described embodiments maybe carried out without departing from the principles of the presentinvention, which is intended to be limited only by the scope of theappended claims as interpreted according to the principles of patentlaw.

1. A vehicular control system, said vehicular control system comprising:a plurality of vehicular cameras disposed at a vehicle equipped withsaid vehicular control system, said plurality of vehicular camerashaving respective fields of view exterior of the equipped vehicle;wherein said plurality of vehicular cameras comprises at least (i) aforward-viewing vehicular camera disposed at the equipped vehicle andhaving a field of view at least forward of the equipped vehicle, (ii) adriver side sideward-viewing vehicular camera disposed at a driver sideof the equipped vehicle and having a field of view at least sideward ofthe driver side of the equipped vehicle and (iii) a passenger sidesideward-viewing vehicular camera disposed at a passenger side of theequipped vehicle and having a field of view at least sideward of thepassenger side of the equipped vehicle; wherein said forward-viewingvehicular camera is disposed in the equipped vehicle behind a windshieldof the equipped vehicle and views forward of the equipped vehiclethrough the windshield; wherein (i) said forward-viewing vehicularcamera of said plurality of vehicular cameras comprises a megapixelimaging sensor having at least one million photosensing pixels, (ii)said driver side sideward-viewing vehicular camera of said plurality ofvehicular cameras comprises a megapixel imaging sensor having at leastone million photosensing pixels and (iii) said passenger sidesideward-viewing vehicular camera of said plurality of vehicular camerascomprises a megapixel imaging sensor having at least one millionphotosensing pixels; a central control module, wherein said centralcontrol module comprises a data processor; wherein said central controlmodule is disposed in the equipped vehicle at a location that is remotefrom the location at the windshield of the equipped vehicle where saidforward-viewing vehicular camera is disposed; wherein said dataprocessor of said central control module comprises an image processor;wherein image data captured by (i) said forward-viewing vehicularcamera, (ii) said driver side sideward-viewing vehicular camera and(iii) said passenger side sideward-viewing vehicular camera is providedto said central control module; a plurality of radar sensors disposed atthe equipped vehicle and sensing exterior the equipped vehicle; whereinradar data captured by said plurality of radar sensors is provided tosaid central control module; wherein said plurality of radar sensorsdisposed at the equipped vehicle and sensing exterior the equippedvehicle comprises a front radar sensor mounted at a front portion of theequipped vehicle; wherein said front radar sensor has a field of sensingat least forward of the equipped vehicle; wherein said central controlmodule receives vehicle data relating to operation of the equippedvehicle; and wherein, during a driving maneuver of the equipped vehicle,and responsive at least in part to fusion at said central control moduleof (i) image data captured by at least said forward-viewing vehicularcamera and (ii) radar data captured by at least said front radar sensor,a pedestrian present exterior of the equipped vehicle is detected. 2.The vehicular control system of claim 1, wherein, responsive at least inpart to processing at said central control module of (i) vehicle data,(ii) image data and (iii) radar data, said central control moduledetermines that presence of the detected pedestrian exterior of theequipped vehicle constitutes a potentially hazardous driving conditionfor the equipped vehicle, and wherein, responsive to said centralcontrol module determining that presence of the detected pedestrianexterior of the equipped vehicle constitutes the potentially hazardousdriving condition for the equipped vehicle, said central control modulecontrols braking of the equipped vehicle.
 3. The vehicular controlsystem of claim 2, wherein, responsive at least in part to processing atsaid central control module of (i) vehicle data, (ii) image data and(iii) radar data, said central control module determines that presenceof the detected pedestrian exterior of the equipped vehicle constitutesa potentially hazardous driving condition for the equipped vehicle, andwherein, responsive to said central control module determining thatpresence of the detected pedestrian exterior of the equipped vehicleconstitutes the potentially hazardous driving condition for the equippedvehicle, said central control module controls steering of the equippedvehicle.
 4. The vehicular control system of claim 3, wherein saidvehicle data comprises at least one selected from the group consistingof (i) vehicle speed data, (ii) vehicle steering data, (iii) vehicle yawrate data and (iv) vehicle acceleration data.
 5. The vehicular controlsystem of claim 1, wherein said central control module controls brakingof the equipped vehicle responsive at least in part to (i) processing atsaid central control module of image data captured by at least saidforward-viewing vehicular camera and (ii) processing at said centralcontrol module of radar data captured by at least said front radarsensor.
 6. The vehicular control system of claim 5, wherein said centralcontrol module controls braking of the equipped vehicle responsive atleast in part to said image processor processing image data captured byat least said forward-viewing vehicular camera and by at least one othervehicular camera of said plurality of vehicular cameras.
 7. Thevehicular control system of claim 6, wherein the at least one othervehicular camera of said plurality of vehicular cameras comprises saiddriver side sideward-viewing vehicular camera.
 8. The vehicular controlsystem of claim 1, wherein said plurality of vehicular cameras providingcaptured image data to said central control module comprises arearward-viewing vehicular camera disposed at the equipped vehicle andhaving a field of view at least rearward of the equipped vehicle, andwherein image data captured by said rearward-viewing vehicular camera isprovided to said central control module, and wherein saidrearward-viewing vehicular camera of said plurality of vehicular camerascomprises a megapixel imaging sensor having at least one millionphotosensing pixels.
 9. The vehicular control system of claim 8, whereinsaid rearward-viewing vehicular camera is disposed at a rear portion ofthe equipped vehicle.
 10. The vehicular control system of claim 1,wherein said central control module comprises a fusion core, and whereinsaid fusion core at least receives (a) radar data captured by said atleast said front radar sensor of said plurality of radar sensors that isprovided to said central control module and (b) image data captured bysaid forward-viewing vehicular camera that is provided to said centralcontrol module, and wherein said central control module receives vehicledata relating to operation of the equipped vehicle and at least oneselected from the group consisting of (i) data associated with a currentgeographic location of the equipped vehicle and (ii) data wirelesslytransmitted to the equipped vehicle.
 11. The vehicular control system ofclaim 1, wherein said central control module comprises a fusion core,and wherein said fusion core is operable to process data for a pluralityof driver assistance systems of the equipped vehicle, and wherein saidplurality of driver assistance systems of the equipped vehicle at leastcomprises (i) a lane keep assist (LKA) system of the equipped vehicle,(ii) a pedestrian detection system of the equipped vehicle, (iii) anautomatic headlamp control (AHC) system of the equipped vehicle and (iv)at least one selected from the group consisting of (a) an adaptivecruise control system of the equipped vehicle (b) a lane departurewarning (LDW) system of the equipped vehicle, (c) a traffic signrecognition (TSR) system of the equipped vehicle, (d) a forwardcollision warning (FCW) system of the equipped vehicle, (e) a vehicledetection (VD) system of the equipped vehicle, (f) a hazard detection(HD) system of the equipped vehicle and (g) an intelligent light ranging(ILR) system of the equipped vehicle.
 12. The vehicular control systemof claim 1, wherein, responsive at least in part to fusion at saidcentral control module of (i) captured image data and (ii) capturedradar data, said central control module at least in part controls (a)braking of the equipped vehicle and (b) steering of the equippedvehicle.
 13. The vehicular control system of claim 12, wherein,responsive at least in part to fusion at said central control module of(i) captured image data and (ii) captured radar data, said centralcontrol module at least in part controls acceleration of the equippedvehicle.
 14. The vehicular control system of claim 1, wherein,responsive at least in part to fusion at said central control module of(i) captured image data and (ii) captured radar data, said centralcontrol module at least in part controls (i) steering of the equippedvehicle and (ii) speed of the equipped vehicle.
 15. The vehicularcontrol system of claim 14, wherein said central control module at leastin part controls an adaptive cruise control system of the equippedvehicle.
 16. The vehicular control system of claim 1, wherein saidplurality of radar sensors comprises a rear-sensing radar sensor mountedat a rear portion of the equipped vehicle and having a field of sensingat least rearward of the equipped vehicle.
 17. The vehicular controlsystem of claim 1, wherein at least one lidar sensor is disposed at theequipped vehicle and senses exterior the equipped vehicle, and whereinlidar data captured by said at least one lidar sensor is provided tosaid central control module, and wherein said central control module atleast in part detects the pedestrian present exterior of the equippedvehicle responsive at least in part to processing at said centralcontrol module of lidar data captured by said at least one lidar sensor.18. The vehicular control system of claim 17, wherein said at least onelidar sensor comprises a three-dimensional sensing lidar sensor.
 19. Thevehicular control system of claim 18, wherein said at least one lidarsensor comprises a scanning lidar sensor.
 20. The vehicular controlsystem of claim 1, wherein said central control module generates anoutput for at least one vehicle control of the equipped vehicle, andwherein said at least one vehicle control comprises at least oneselected from the group consisting of (i) a vehicle body control, (ii) avehicle chassis control, (iii) a vehicle supervisory control, (iv) avehicle brake control, (v) a vehicle steering control and (vi) a vehicleengine control.
 21. The vehicular control system of claim 1, whereinsaid central control module, responsive to a geographical location ofthe equipped vehicle, at least in part controls a plurality of driverassistance systems of the equipped vehicle, said plurality of driverassistance systems of the equipped vehicle at least comprising (i) alane keep assist (LKA) system of the equipped vehicle, (ii) a pedestriandetection system of the equipped vehicle, (iii) an automatic headlampcontrol (AHC) system of the equipped vehicle and (iv) at least oneselected from the group consisting of (a) an adaptive cruise controlsystem of the equipped vehicle (b) a lane departure warning (LDW) systemof the equipped vehicle, (c) a traffic sign recognition (TSR) system ofthe equipped vehicle, (d) a forward collision warning (FCW) system ofthe equipped vehicle, (e) a vehicle detection (VD) system of theequipped vehicle, (f) a hazard detection (HD) system of the equippedvehicle and (g) an intelligent light ranging (ILR) system of theequipped vehicle.
 22. The vehicular control system of claim 1, whereinimage data captured by said forward-viewing vehicular camera is providedto said central control module via an Ethernet link.
 23. The vehicularcontrol system of claim 1, wherein said central control module comprisesa vision core, and wherein said plurality of vehicular cameras comprisesa rearward-viewing vehicular camera disposed at a rear portion of theequipped vehicle, and wherein said rearward-viewing vehicular camera ofsaid plurality of vehicular cameras comprises a megapixel imaging sensorhaving at least one million photosensing pixels, and wherein image datacaptured by said rearward-viewing vehicular camera is provided to saidcentral control module, and wherein said driver side sideward-viewingvehicular camera is disposed at a driver-side exterior rearview mirrorassembly of the equipped vehicle, and wherein said passenger sidesideward-viewing vehicular camera is disposed at a passenger-sideexterior rearview mirror assembly of the equipped vehicle, and whereinbased on processing at said vision core of image data captured at leastby (i) said rearward-viewing vehicular camera, (ii) said driver sidesideward-viewing vehicular camera and (iii) said passenger sidesideward-viewing vehicular camera, a bird's eye view of a regionexterior the equipped vehicle is generated and is output by said centralcontrol module for displaying on a video display screen of the equippedvehicle.
 24. The vehicular control system of claim 1, wherein said imageprocessor processes image data captured by at least said forward-viewingvehicular camera to determine a driving situation and wherein, indetermining said driving situation, a fusion core of said centralcontrol module processes sensor data captured by said front radar sensorto augment determination of the driving situation.
 25. The vehicularcontrol system of claim 1, wherein said vehicular control system,responsive at least in part to processing at said central control moduleof image data captured by at least said forward-viewing vehicularcamera, at least in part controls a plurality of driver assistancesystems of the equipped vehicle, said plurality of driver assistancesystems of the equipped vehicle at least comprising (i) a lane keepassist (LKA) system of the equipped vehicle, (ii) a pedestrian detectionsystem of the equipped vehicle, (iii) an automatic headlamp control(AHC) system of the equipped vehicle and (iv) at least one selected fromthe group consisting of (a) an adaptive cruise control system of theequipped vehicle (b) a lane departure warning (LDW) system of theequipped vehicle, (c) a traffic sign recognition (TSR) system of theequipped vehicle, (d) a forward collision warning (FCW) system of theequipped vehicle, (e) a vehicle detection (VD) system of the equippedvehicle, (f) a hazard detection (HD) system of the equipped vehicle and(g) an intelligent light ranging (ILR) system of the equipped vehicle.26. The vehicular control system of claim 1, wherein said imageprocessor of said central control module comprises an image processingchip, and wherein image data captured by said forward-viewing vehicularcamera is provided to said central control module via a digital signalcarried over a wired connection between said forward-viewing vehicularcamera and said central control module.
 27. The vehicular control systemof claim 26, wherein said plurality of radar sensors disposed at theequipped vehicle and sensing exterior the equipped vehicle comprises aleft-side radar sensor mounted at a left side of the equipped vehicleand a right-side radar sensor mounted at a right side of the equippedvehicle.
 28. The vehicular control system of claim 27, wherein saidplurality of radar sensors disposed at the equipped vehicle and sensingexterior the equipped vehicle comprises a rear radar sensor mounted at arear portion of the equipped vehicle.